Computing Similarity between Cultural Heritage Items using Multimodal Features

In recent years a vast amount of Cultural Heritage (CH) artefacts have been digitised and made available on-line. For example, the Louvre and the British Museum provide information about exhibits on their web pages1. In addition, information is also available via sites that aggregate CH information from multiple resources. A typical example is Europeana2, a web-portal to collections from several European institutions that provides access to over 20 million items including paintings, films, books, archives and museum exhibits. However, online information about CH artefacts is often unstructured and varies by collection. This makes it difficult to identify information of interest in sites that aggregate information from multiple sources, such as Europeana, or to compare information across multiple collections (such as the Louvre and British Museum). These problems form a significant barrier to accessing the information available in these online collections. A first step towards improving access would be to identify similar items in collections. This could assist with several applications that are of interest to those working in CH including recommendation of interesting items (Wang et al., 2008), generation of virtual tours (Wang et al., 2009), visualisation of collections (Hornbaek and Hertzum, 2011) and exploratory search (Amin et al., 2008). Information in digital CH collections often includes multiple types of media such as text, images and audio. It seems likely that information from all of these types would help humans to identify similar items and that it could help to identify them automatically. In this work we make use of information from two media (text and images) to compute the similarity between items in a large collection of CH items. A data set was created by selecting 295 pairs of items from Europeana. Each item corresponds to a metadata record consisting of textual information together with a URI and a link to its thumbnail. Figure 1 shows an item taken from the Europeana website. These are extracted and concatenated to form the textual information associated with each item. In addition, the accompanying thumbnail image (or “preview”) was also extracted to be used as the visual information. The size of these images varies from 7,000 to 10,000 pixels. A range of similarity measures for text and images are compared and combined. Text similarity is measured by applying a range of corpus-based methods, i.e. tf.idf and LDA (Blei et al., 2003). Image similarity is computed by making use of colour histograms and the Haar wavelet decomposition (Beylkin et al., 1991) to create signatures of images that contain colour and basic shape information. Evaluation is carried out using a set of items from Europeana with similarity judgements that were obtained in a crowdsourcing experiment. We find that combining information from both media produces better results than when either is used alone. http://www.louvre.fr/, http://www.britishmuseum.org/ http://www.europeana.eu